Skip to content

[core] Reduce memory for global index-only commits - #8896

Draft
leaves12138 wants to merge 1 commit into
apache:masterfrom
leaves12138:agent/optimize-index-only-commit-memory
Draft

[core] Reduce memory for global index-only commits#8896
leaves12138 wants to merge 1 commit into
apache:masterfrom
leaves12138:agent/optimize-index-only-commit-memory

Conversation

@leaves12138

@leaves12138 leaves12138 commented Jul 29, 2026

Copy link
Copy Markdown
Contributor

What changed

  • add a dedicated low-memory conflict-check path for global index-only commits
  • scan projected BinaryManifestEntry fields directly instead of materializing all data files as SimpleFileEntry
  • prune manifest files with partition statistics and manifest row-id bounds
  • use reusable byte-array partition lookups, compact deleted-file identifiers, and primitive row-range storage
  • preserve existing data manifests for index-only commits instead of opportunistically sorting or merging them
  • retain the original conflict path for data-file changes, deletion vectors, commit pre-callbacks, and rowIdCheckFromSnapshot

Why

A global index-only commit previously read every live data entry in the affected partitions into Java objects and retained that base-file list through snapshot preparation. It also invoked data-manifest sorting even though no data file changed.

On the large test table, conflict detection materialized 4,105,298 SimpleFileEntry instances and the subsequent manifest sort rewrote 32 data manifests. These phases overlapped in memory.

The new path still validates that every global-index row-id range is covered by current live data files. It performs the check with projected binary manifest entries, applies DELETE entries by full file identity, and retains only primitive ranges intersecting the requested index ranges.

Measured impact

Snapshot 3496 metadata, Linux x86, -Xmx24g:

Metric Pre-change (bbcac771ff) This change, rebased on 903aa197a1
Commit elapsed time 64.127 s 3.102 s
Peak Java heap 16.384 GiB 0.899 GiB
Peak process RSS 19.160 GiB 1.492 GiB

The index commit produced the next snapshot successfully, and the tests assert that its data-manifest list is unchanged.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant